Restricted Boltzmann Machines as Models of Interacting Variables

Abstract We study the type of distributions that restricted Boltzmann machines (RBMs) with different activation functions can express by investigating effect function hidden nodes on marginal distribution they impose observed binary nodes. report an exact expression for these marginals in form a model interacting variables explicit interactions depending node function. properties detail and evaluate how accuracy which RBM approximates over depends number When inferred parameters are weak, intuitive pattern is found interaction terms, reduces substantially differences across functions. show weak parameter approximation good RBMs trained MNIST data set. Interestingly, cases, mapping reveals models essentially low order models.